{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Sven Kreiss](https://www.svenkreiss.com/), 2020\n",
    "\n",
    "# Training\n",
    "\n",
    "This section introduces training to create your own models. You don't need to do this step \n",
    "if you use pre-trained models for {doc}`predict_cli` on your own images.\n",
    "For training, you will need a dataset. See {doc}`datasets` for instructions about a few popular datasets.\n",
    "\n",
    "Training a model can take several days even with a good GPU. Many times existing models can be refined to avoid training from scratch.\n",
    "\n",
    "A quick way to get started is with the training commands of the pre-trained models.\n",
    "The exact training command that was used for a model is in the first\n",
    "line of the training log file. Below are a few examples for various backbones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ShuffleNet\n",
    "\n",
    "ShuffleNet models are trained without ImageNet pretraining:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```sh\n",
    "time CUDA_VISIBLE_DEVICES=0,1 python3 -m openpifpaf.train \\\n",
    "  --lr=0.0001 --momentum=0.98 --b-scale=10.0 \\\n",
    "  --epochs=150 \\\n",
    "  --lr-decay 130 140 \\\n",
    "  --lr-decay-epochs=10 \\\n",
    "  --batch-size=32 \\\n",
    "  --square-edge=385 \\\n",
    "  --weight-decay=1e-5 \\\n",
    "  --update-batchnorm-runningstatistics \\\n",
    "  --basenet=shufflenetv2k16w \\\n",
    "  --headnets cif caf caf25\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For improved performance, take the epoch150 checkpoint and train with\n",
    "extended-scale and 10% orientation invariance:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```sh\n",
    "time CUDA_VISIBLE_DEVICES=0,1 python3 -m openpifpaf.train \\\n",
    "  --lr=0.00005 --momentum=0.98 --b-scale=10.0 \\\n",
    "  --epochs=250 \\\n",
    "  --lr-warm-up-start-epoch=150 \\\n",
    "  --lr-decay 220 240 \\\n",
    "  --lr-decay-epochs=10 \\\n",
    "  --batch-size=32 \\\n",
    "  --square-edge=385 \\\n",
    "  --weight-decay=1e-5 \\\n",
    "  --update-batchnorm-runningstatistics \\\n",
    "  --checkpoint outputs/shufflenetv2k30w-200811-232815-cif-caf-caf25-659c5af6.pkl --extended-scale --orientation-invariant=0.1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can refine an existing model with the `--checkpoint` option.\n",
    "\n",
    "For large models, reduce the batch size and learning rate by the same factor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ResNet\n",
    "\n",
    "ResNet models are initialized with weights pre-trained on ImageNet.\n",
    "That makes their training characteristics different from ShuffleNet (i.e. they look great at the beginning of training)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logs\n",
    "\n",
    "To visualize logs:\n",
    "\n",
    "```sh\n",
    "python3 -m openpifpaf.logs \\\n",
    "  outputs/resnet50block5-pif-paf-edge401-190424-122009.pkl.log \\\n",
    "  outputs/resnet101block5-pif-paf-edge401-190412-151013.pkl.log \\\n",
    "  outputs/resnet152block5-pif-paf-edge401-190412-121848.pkl.log\n",
    "```\n",
    "\n",
    "To produce evaluation metrics every five epochs and check the directory for new\n",
    "checkpoints every 5 minutes:\n",
    "\n",
    "```sh\n",
    "while true; do \\\n",
    "  CUDA_VISIBLE_DEVICES=0 find outputs/ -name \"shufflenetv2k16w-200504-145520-cif-caf-caf25.pkl.epoch??[0,5]\" -exec \\\n",
    "    python3 -m openpifpaf.eval_coco --checkpoint {} --long-edge=641 --skip-existing \\; \\\n",
    "  ; \\\n",
    "  sleep 300; \\\n",
    "done\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": 3
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
